Discrete Time Markov Chains
Markov processes
We have seen that sequences are easiest discrete time random processes. However, they don’t capture correlation well.
Definition 1. A stochastic process X:\Omega\to\sX^T with state space \sX and ordered index set T is Markov if conditioned on the present state X_t, future \sigma(X_u, u > t) is independent of the past \sigma(X_s, s< t). We denote the history of the process until time t as \sF_t \triangleq \sigma(X_s, s \le t). That is, for any Borel measurable set B \in \sB(\sX) any two indices u > t, we have
Remark 1. We next re-write the Markov property more explicitly for the process X:\Omega\to\R^\R. For all x, y \in \sX, finite set S \subseteq \R such that \max S < t < u, and H_S(x_S) \triangleq \cap_{s \in S}\set{X_s \le x_s} \in \sF_t, we have
Discrete time Markov chains
Definition 2. For a state space \sX \subseteq \R and the random sequence X: \Omega \to \sX^{\Z_+}, we define the history until time n \in \Z_+ as \sF_n \triangleq \sigma(X_1, \dots, X_n).
Remark 2. Recall that the event space \sF_n is generated by the historical events A_X(x) \triangleq \cap_{i=0}^n\set{X_i \le x_i} where x \in \R^{n+1}.
Remark 3. When the state space \sX is countable, the event space \sF_n is generated by the historical events H_n(x) \triangleq \cap_{i=0}^n\set{X_i = x_i}, where x \in \sX^{n+1}. That is, \sF_n = \sigma(H_n(x):x \in \sX^{n+1})
Definition 3. For a countable set \sX, a discrete-valued random sequence X: \Omega \to \sX^{\Z_+} is called a discrete time Markov chain (DTMC) if for all positive integers n \in \Z_+, all states x,y \in \sX, and any historical event H_{n-1} = \cap_{m=0}^{n-1}\set{X_m = x_m} \in \sF_n for (x_0, \dots, x_{n-1}) \in \sX^{n}, the process X satisfies the Markov property
Remark 4. The above definition is equivalent to P(\set{X_{n+1} \le x}\given \sF_n) = P(\set{X_{n+1} \le x}\given \sigma(X_n)) for discrete time discrete state space Markov chain X, since \sF_n = \sigma(H_n(x): x \in \sX^n) and \sigma(X_n) = \sigma(\set{X_n = x}, x \in \sX).
Example 4 (Random Walk). A random walk S:\Omega\to\sX^\N with independent step-size sequence X:\Omega\to\sX^\N, is Markov for a countable state space \sX that is closed under addition. Given a historical event H_{n-1}(s) \triangleq \cap_{k=1}^{n-1}\set{S_k = s_k} and the current state \set{S_n = s_n}, we can write the conditional probability The equality in the second line follows from the independence of the step-size sequence. In particular, from the independence of X_{n+1} from the collection \sigma(S_0, X_1, \dots, X_n) = \sigma(S_0, S_1, \dots, S_n).
Transition probability matrix
Definition 5. We denote the set of all probability mass functions over a countable state space \sX by \cM(\sX)\triangleq \set{\nu \in [0,1]^\sX: \sum_{x\in \sX}\nu_x = 1}.
Definition 6. The transition probability matrix at time n is denoted by P(n) \in [0,1]^{\sX \times \sX}, such that its (x,y)th entry is denoted by p_{xy}(n) \triangleq P(\set{X_{n+1} = y}\given\set{X_n = x}), that is the transition probability of a discrete time Markov chain X from a state x \in \sX at time n to state y \in \sX at time n+1.
Remark 5. We observe that each row P_{x}(n) \triangleq (p_{xy}(n): y \in \sX) \in\cM(\sX) is the conditional distribution of X_{n+1} given the event \set{X_n = x}.
Definition 7. A matrix A \in \R_+^{\sX \times \sX} with non-negative entries is called sub-stochastic if the row-sum \sum_{y \in \sX} a_{xy} \le 1 for all rows x \in \sX. If the above property holds with equality for all rows, then it is called a stochastic matrix. If matrices A and A^T are both stochastic, then the matrix A is called doubly stochastic.
Remark 6. We make the following observations for the stochastic matrices.
Every probability transition matrix P(n) is a stochastic matrix.
All the entries of a sub-stochastic matrix lie in [0,1].
Each row A_x \triangleq (a_{xy}: y \in \sX) of the stochastic matrix A \in \R_+^{\sX \times \sX} belongs to \cM(\sX).
Every finite stochastic matrix has a right eigenvector with unit eigenvalue. This can be observed by taking \bU^T = \begin{bmatrix}1 & \dots & 1\end{bmatrix} to be an all-one vector of length \abs{\sX}. Then we see that A \bU = \bU, since
Every finite doubly stochastic matrix has a left and right eigenvector with unit eigenvalue. This follows from the fact that finite stochastic matrices A and A^T have a common right eigenvector \bU. It follows that A has a left eigenvector \bU^T.
For a probability transition matrix P(n), we have \sum_{y \in \sX}f(y)p_{xy}(n) = \E[f(X_{n+1})\mid\set{X_n =x}].
Homogeneous Markov chains
In general, not much can be said about Markov chains with index dependent transition probabilities. We consider the simpler case where the transition probabilities p_{xy}(n)= p_{xy} are independent of the index.
Definition 8. A discrete time Markov chain with the probability transition matrix P(n) that is independent of the index, is called time homogeneous.
Example 9 (Integer random walk). For a one-dimensional integer valued random walk S: \Omega \to \Z^\N with unit step size sequence X: \Omega \to \set{-1,1}^\N such that P\set{X_1 = 1} = p, the transition operator P \in [0,1]^{\Z\times\Z} is given by the entries p_{xy} = p\indicator{y=x+1}+(1-p)\indicator{y=x-1} for all x,y \in\Z.
Example 10 (Sequence of experiments). Consider a random sequence of experiment outcomes X: \Omega\to\set{0,1}^{\Z_+}, such that P(\set{X_{n+1} = 0}\mid\set{X_n=0})= 1-q and P(\set{X_{n+1} = 1}\mid\set{X_n=1})= 1-p for all n \in \Z_+. Then, we can write the probability transition matrix as
Definition 11. Consider a time homogeneous Markov chain X: \Omega \to \sX^{\Z_+} with countable state space \sX and transition matrix P. We would respectively denote the conditional probability of events and conditional expectation of random variables, conditioned on the initial state \set{X_0 = x}, by
Proposition 12. Conditioned on the initial state, any finite dimensional distribution of a homogeneous Markov chain is stationary. That is, for any finite n,m \in \Z_+ and states x_0, \dots, x_n \in \sX, we have
Proof. Proof. Consider a homogeneous Markov chain X:\Omega\to\sX^{\Z_+} with natural filtration \sF_\bullet such that \sF_n \triangleq \sigma(X_0, \dots, X_n) for each n\in\N. Using the property of conditional probabilities and Markovity of X, we can write the conditional probability of sample path (X_1, \dots, X_n) given the event \set{X_0 = x_0} as Using the property of conditional probabilities and Markovity of X, we can write the conditional probability of sample path (X_{m+1}, \dots, X_{m+n}) given the event \set{X_m = x_0} as From time-homogeneity of transition probabilities of Markov chain X, it follows that both the transition probabilities are identical and equal to \prod_{i=1}^np_{x_{i-1},x_i}. ◻
Corollary 13. The n-step transition probabilities are stationary for any homogeneous Markov chain. That is, for any states x_0,x_n \in \sX and n,m \in \N, we have P(\set{X_{n+m} = x_n}\mid\set{ X_m = x_0}) = P(\set{X_{n} = x_n}\mid\set{ X_0 = x_0}).
Proof. Proof. It follows from summing over all possible paths (X_0, \dots, X_n) and (X_m, \dots, X_{m+n}). In particular, we can partition events \set{X_n=x_n} and \set{X_{m+n}=x_n} in terms of unions over disjoint paths The result follows from the countable additivity of conditional probability for disjoint events of taking distinct paths, and the fact that probability of taking same path is identical for both sums. ◻
Transition graph
A time homogeneous Markov chain X: \Omega\to\sX^\N with a probability transition matrix P, is sometimes represented by a directed weighted graph G = (\sX, E, w), where the set of nodes in the graph G is the state space \sX, and the set of directed edges is the set of possible one-step transitions indicated by the initial and the final state, as In addition, this graph has a weight w_e = p_{xy} on each edge e = [x,y\rangle \in E.
Example 14 (Integer random walk). The time homogeneous Markov chain in Example [exmp:IRW] can be represented by an infinite state weighted graph G = (\Z, E, w), where the edge set is We have plotted the sub-graph of the entire transition graph for states \set{-1,0,1} in Figure 1.
Example 15 (Sequence of experiments). The time homogeneous Markov chain in Example [exmp:MarkovBerExp] can be represented by the following two-state weighted transition graph G = (\set{0,1}, E, w), plotted in Figure 2.
Random mapping theorem
We saw some example of Markov processes where X_n = X_{n-1} + Z_n, and Z:\Omega\to\Lambda^\N is an sequence, independent of the initial state X_0. We will show that any discrete time Markov chain is of this form, where the sum is replaced by arbitrary functions.
Theorem 16 (Random mapping theorem). For any DTMC X:\Omega\to\sX^{\Z_+}, there exists an sequence Z \in \Lambda^\N and functions f_n: \sX \times \Lambda \to \sX such that X_{n} = f_n(X_{n-1}, Z_{n}) for all n \in \N.
Remark 7. A random mapping representation of a transition matrix P(n) on state space \sX is a function f_n: \sX \times \Lambda \to \sX, along with a random variable Z_n:\Omega\to\Lambda, satisfying for all x,y \in \sX,
Proof. Proof. It suffices to show that every transition matrix P(n) has a random mapping representation. Then, for the mapping f_n and the sequence Z:\Omega\to\Lambda^\N, we would have X_n = f_n(X_{n-1}, Z_n) for all n \in \N.
Let \Lambda \triangleq [0,1], and we choose the uniform sequence Z:\Omega\to\Lambda^\N. Since \sX is countable, it can be ordered. We let \sX = \N without any loss of generality. We set F_{xy}(n) \triangleq \sum_{w \le y}p_{xw}(n) and define function f_n:\sX\times\Lambda\to\sX for all pairs (x,z)\in\sX\times\Lambda by Since f_n(x,Z_n) is a discrete random variable taking value y \in \sX, iff the uniform random variable Z_n lies in the interval (F_{x, y-1}(n), F_{x,y}(n)]. That is, the event \set{f_n(x,Z_n) = y} = \set{Z_n \in (F_{x, y-1}(n), F_{x,y}(n)]} for all y \in \sX. It follows that ◻